Static and Dynamic Neural Networks

As computers become more and more powerful, as algorithms improve, more and more computing power is behind the engineering systems’ behavior. As a result of this increasing sophisticated data processing and decision making, many current systems exhibit more and more intelligent behavior. However, no matter how intelligent they may seem, in comparison with the previous less sophisticated systems, true intelligence also means learning, and many existing systems, from many sophisticated robots to automatic control systems for power grid, do not have the ability to learn. One reason for this is the fact that learning is difficult. For simple static systems, when the output y depends on the few inputs x1, . . . , xn, we can learn the appropriate dependence y = f(x1, . . . , xn), e.g., by using artificial neural networks that use backpropagation or a similar rule to learn. For many real life systems, the situation is not so easy. For example, for controlling a power grid, we must make decisions y(t) based not only on the current values xi(t) of power production and power demands, but also on the past values x i(t′) (t′ < t) of the corresponding parameters. The objective is to maximize the discounted utility ∑ q · u(t), where u(t) is the overall utility at time t and q < 1 is a discounting coefficient. To describe a general control of such a system, we must describe the dependence of the current control parameter y(t) on many different variables xi(t), xi(t − 1), etc. With a large number of inputs, not only the neural network training becomes difficult, but, as Paul Werbos, the author of backpropagation, has shown, in such case, often, backpropagation leads to an irrelevant local minimum instead of the correct values of the weights. How can we make training feasible for such dynamic systems? One problem is that we have both uncertainty in how the system would react to different controls – the uncertainty that requires neural network training to find this out – and the complex dynamic character of the system. We have already mentioned that when the system is not complex, the problem becomes efficiently solvable – we can use traditional neural networks to train it. In the other simplified case, when the system is dynamic but its behavior is known, the problem also becomes efficiently solvable: namely, we can use dynamic programming to solve the corresponding optimization problem. It is therefore desirable to try to combine the methods for solving these two simplified cases, i.e., neural networks and dynamic programming, to handle the optimization of a dynamic system in the situation when the behavior of this system is not known exactly and need to be learned. This combination is not as wishful as it may sound because the backpropagation methods for training neural networks and dynamic programming methods for optimizing nonlinear systems actually have an idea in common. Indeed, the main idea behind dynamic programming is that to optimize the decision making, at a given moment to time t, we select a control u(t) that optimizes a special auxiliary function J(t, x(t)). To be more precise, we first go “backwards” in time and compute the value of the function J(t, x) for t = T, T − 1, . . .; after this, we go forward in time and compute the controls u(0), u(1), . . . A similar alternation of forward and backward steps is what makes backpropagationcomputationally efficient: first, we go forward, from the input to the output, and use the current weights to compute the output; then, we compare the results with the desired results and use these errors to propagate back to the inputs and update the weights along the way. This handbook appeared as a result of the workshop organized in 2002 by the National Science Foundation to enhance the collaboration between dynamic programming and computer learning communities. This was the first serious joint meeting of the two research communities that really boosted the collaboration. The papers resulting from this collaboration form this handbook.